Performance of distributed applications can depend heavily on the behavior of underlying networks. Nevertheless, the problem of understanding how well various Internet Service Provider (ISP) networks deliver traffic has received scant attention. To date, customers of these networks have typically been in the dark about which Internet Service Providers (ISPs) are better than others, or if a higher price necessarily provides indication of better performance. A common method for customers to obtain this information has been by asking each other about their respective experiences with certain providers. Similarly, distributed applications are equally unaware of how their performance can be impacted by their choice of Internet Service Provider (ISP) (e.g., when available, for instance, with DNS redirectors or overlays), unless it is actively measured.
In addition to enabling better distributed applications and helping customers, shedding light on the relative performance of Internet Service Providers (ISPs) can also improve network infrastructure as a whole. The overall performance of the Internet depends collectively on multiple Internet Service Providers (ISPs), and the inability to differentiate between individual Internet Service Providers (ISPs) discourages innovation and does not provide strong incentives to resolve problems. To address this, some researchers have proposed radically different (and arguably heavy-weight mechanisms) based on Internet Service Provider (ISP) accountability and overlay or customer directed routing. Other researchers in contrast have posited that by merely providing visibility into the relative performance of Internet Service Providers (ISPs) provides the right incentives for Internet Service Providers to adopt actions to correct deficiencies in their networks. For instance, no Internet Service Provider (ISP) is necessarily motivated to improve its network where it is merely proclaimed that the average latency in the Internet is 60 ms. However, if it is announced that the average latency for customers of one Internet Service Provider (ISP) is 20 ms whereas the average latency of another Internet Service Provider (ISP) is 200 ms, market forces will likely motivate the second Internet Service Provider (ISP) to correct whatever is inducing this comparative disadvantage.
The relative performance of Internet Service Providers (ISPs) can depend on several factors, including the distance between the source and the destination, the geographic properties of traffic, and even time of day. Further, the performance of paths internal to an Internet Service Provider (ISP), which can form the basis of typical Service Level Agreements (SLAs) and the commercial effort, may not directly reflect end-to-end performance. Thus, the choice of an Internet Service Provider (ISP) can be a complex decision requiring a detailed analysis of Internet Service Provider (ISP) performance.
Based on measurements of their own network, many Internet Service Providers (ISPs) offer a Service Level Agreement (SLA) that specifies the performance that customers can expect. But perhaps because of their unwillingness to vouch for performance outside their network, these Service Level Agreements (SLAs) are typically not end-to-end and mention performance only within the Internet Service Provider's (ISP's) network. For instance, a Service Level Agreement (SLA) may promise that 95% of traffic will not experience a latency of more than 100 ms inside the Internet Service Provider's (ISP's) network. A few providers also offer “off-net” Service Level Agreements (SLAs) in which performance is specified across two networks—the Internet Service Provider's (ISP's) own network and that of some of its neighbors.
For comparing Internet Service Providers (ISPs), current Service Level Agreements (SLAs) have two shortcomings. First, application performance depends on the entire path to the destination and not only on a subpath. As such, Internet Service Providers (ISPs) with better Service Level Agreements (SLAs) may not offer better performance. Second, because they are independently offered by different Internet Service Providers (ISPs), Service Level Agreements (SLAs) make comparisons among Internet Service Providers (ISPs) difficult. Some Service Level Agreements (SLAS) may mention latency, some may mention loss rates, some may mention available capacity, and yet others may mention a combination. Even with comparable measures, difficulties in comparison can stem from the differences in the size and the spread of different Internet Service Providers (ISPs). For instance, is a 100 ms performance bound for an Internet Service Provider (ISP) with an international network necessarily better or worse than a 50 ms bound for an Internet Service Provider (ISP) with only a nationwide network?
Many listings compare broadband and dial-up Internet Service Providers (ISPs) based primarily on their prices and maximum theoretical capacity. For example, one system measures latency and loss rate for paths internal to Internet Service Providers (ISPs) and for paths between pairs of Internet Service Providers (ISPs). This is generally done by co-locating nodes within some Internet Service Providers' (ISPs') Points of Presence (PoPs) and measuring the paths between them. Such an approach however can have several limitations. First, because the approach requires active cooperation from Internet Service Providers (ISPs) to place nodes inside their Points of Presence (PoPs), coverage of an Internet Service Provider's (ISP's) network can be poor. Second, the technique does not typically measure the entire path to the destination but only a part of it. Third, the method probes use of the IP addresses of its measurement nodes as the destination address. Because Internet routing is destination-based, the performance experienced by destination bound traffic may differ from measurement-node-bound traffic.